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Foreword 

This monograph, based on a paper-fry Dr. William River^' 
ror the 1982 Comparative and International Edtftration Society' 
Annual Conference, highlights issues in contemporary iarge-rScale 
evaluation in adult education. I * / 

Evaluation at the international level, asNvell as] at the na- 
tional level, adopts systems approaches ajad-ecpnomic values 
that impose utilitarian purposes on aduy education to the detri- 
ment of other aimss^Rivera's paper, iryresponding to these 
positivist developments in education/ undertake three -tasks. The 
Hrst is to review the differing stance^ taken by spokespersons in 
the international field of adult* education, comparative study, 
and evaluative research on such issues as the dehnition, present 
status, and Kiture trends of evaluation in large-scale, interna- 
tional educational program development. Secondly, an educa- 
tional planning-evaluation symbiosis is posited and then exam- 
ined with reference to the contemporary' focus on education for - 
manpower development.- At the same time, efficiency (cost- 
er fee tiveness) evaluation is criticized as an inadequate criterion 
ror judging the worth of educational programs. Finally, the im- 
plications of leadership in large-scale evaluation being taken over 
by governmental and "nonproht" organizations are considered 
against the role and purpose of the universities, professional 
associations, and academics in this area of applied research. 

Dr. Rivera s analysis of the state of the art in large-scale in- 
ternational evaluation raises a number of issues and questions 
that should be given high, priority by members of the academic 
.community and professional associations in adult education, as 
well as by'pohcy makers, planners, and administrators of adult 
education programs. „ 

Dr. Rivera has worked with several international organiza- 
tions, including the United Nations Educational, Scientific, and 
Cultural Organization (UNESCO), Organization for Economic 
Cooperation and Development (OECD), and World Education, 
Inc. In 197o he served as US. Representative to UNESCO to 
assist. in drafting the international Recommendation on the De- 
velopment of Aciult Education At present Dr. Rivera is Associ- 
ate Professor of Adult Education at The University of Maryland 

Felix Adam and George "F. Aker 
Series Editors 
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National development is a wider and more complex term 
than merely economic growth, and education is only one of the 
major components in the concept of national ^development, ac- 4 
cording to the! IVth Congress of the World Cbuncir of Com : 
parative Elation Societies (WCCES, 1981). Hence, the issue is 
not one ot education fror national development but what educa- 
tion can do in the strategy for national development. 

Education's role in national development is nevertheless 
a large one, and-tbis is apparent from the major investments, 
particularly by developing countries, in education— investments 
often averaging 20 percent and sometimes exceeding one-third of 
national budgets. Today, education's role is recognised as going 
beyond schooling and formal instruction for children and youth; 
it extends to nonformal education activities and especially educa- 
tion tor adults. 

Adult education, broadly conceived as including all types 
and levels ot educational programs for adults, is a significant in- 
put for meeting the development challenge. It stands out as a 
special priority among nations faced with the pressures to pro- 
duce more in the present, that can't wait for children' to grow 
up and economically produce in the future. Educational invest- 
ment in adults and participation of adults in education are both 
on the rise. Some nations have afforded adult education priority 
status within their systems of education; others are considering 
formalization of adult education into a fhird system or fourth 
level, and still others conceive of education for adults as part of 
an integrated lifelong-education complex. 

, Since the 1950s and 1960s the scope and purpose of 
education, including adult education-, have both widened and 
narrowed— widened in the extent of concern and planning but 
narrowed in regard to focusing on predominantly economic 
goals. With wide-range and long-term planning have come large- 
scale program development and, almost inevitably, large-scale 
evaluation. An example, and exemplary case, of large-scale pro- 
gram development and evaluation is the United Nations Educa- 
tional' Scientific, and Cultural Organization/United Nations 
Development Programme (UNESCO/UNDP) Experimental 
World Literacy Programme (EWLP), which the present paper 
reviews and critiques by way of discussing the breadth and 
direction of large-scale evaluative research an$d its implications 
for academics, their professional associations, and universities in 
v^neral. 



The present paper asks a number of questions: What 
are the. differing perceptions of evaluation at the international 
level? Is there a symbiosis between edutational planning and 
evaluation? What are the criteria underlying international educa- 
tionaPevaluations? What types»of evaluation are truly educa- 
tional in nature? Are efficiency evaluations indicative of educa- 
tional success (or failure)? 'What are the implications of large- 
scale planning and evaluation for universities, professional 
associations, ^and academics? 

The paper is organized into three main parts concerning 
perspectives, criteria, and implications of evaluation at the inter- 
national level. The first part reviews definitions, approaches, and 
perspectives of evaluation. The second part examines the educa-^ 
tional pfenning-evaluation symbiosis and its underlying socio- 
economic criteria The thnrd part undefscores the implications of * 
large-scale evaluative research in adult education for universities, 
professional associations, and academics and proposes a number 
of steps for meeting the challenge of leadership that large-scale i 
evaluation presents. 

While the paper takes a critical look at developments in 
large-scale evaluative reseafrch and academe's capacity to meet 
the challenge, its ultimate purpose is to set the stage for an era 
of extensive cooperation among universities^ professional associa- 
tions, and academics concerned with the expanding critical area 
of evaluative research in adult and* comparative education. 




PERSPECTIVES OF EVALUATION IN 
EDUCATION FOR ADULTS ' 

, ) DEFINITIONS 

The intimate Connection between evaluation and planning 
is made evident when onv considers that the very definition of 
evaluation depends upon one's general philosophy of evaluation 
and how one intends to use the acquired evaluation data (Tyler, ' * 
19o9). What is the purpose and goal of the evaluation? What is 
the hierarchical level at which the evaluation will take place? 
Wh^t type of evaluation will be undertaken? - v ^ 

Evaluation me^ns different things to different people,' and 
_ though some would restrict the term to one definition or 
. another, most writers on the subject take an eclectic stance and 
^ . include various meanings (House v 1978, Grotelueschen, 1980; 
* Stake, i074j Stufflebeam, 1977; Taylor, 1976, and Worthen and 
Sanders, J. "However, even eclecticism results in distinct 
perspectives. 

In part, the problem of definition is historical in nature. 
Until recently, evaluation when applied to education meant the' 
measurement of student achievement With expansion and ag- 
grandizement of educational programs in the 1960s and 70s, 
however, operational questions became paramount and new 
evaluative concerns came to the fore. Indeed, a major dividing 
line can be traced to this period of program expansion when 
classical evaluation of student performance and the teaching 
situation (methods, techniques, and materials) began to be over- 
powered by program operational evaluation concerned with 
cost-effectiveness and social outcomes. ' • . 

\ Even today, in the early 1980s, the major dividing line 

between classical, and operational evaluation is only now becom- 
ing-clear. Evaluation stijl tends to bfe defined as though^ this 
dividing line didn't exjst. and specialists speak of evaluation as a 
time-onenteJ process taking place before program initiation (in- # 
volving needs assessment and institution-building analyses); mid- 
course when the project is underway (formative evaluation); and 
at the end of c a program or program cycle {summative evalua- 
tion). Oc else. Evaluation is considered as a process with dif- , 
ferent purposes, such as: 

—documenting the congruence of k^amer outcomes and 
program objectives (Tyler, 1950) 

r >■ 
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—comparing pert6rmance data with a commonly accepted 
' Standard (Popham., 1969; Provus, 1969; Rivlin, 1971) > 
—specifying, obtaining, and providing relevant information 
for judging decision alternatives (Alkin, 1967; 
Stufflebeam, 1969) , 
, —comparing actual effects of a pjogram with a variety of 

demonstrated needs (Scriven, 1972) 
—judging program merit against the value positions of rele- 
0 vant audiences (McDonald, 1974; Owens, 1973; Rippey, 
.1973; Stake, 1974) 
— descnbing and interpreting the wider context in which a 

program functions (Parlett and Hamilton, 1976) 
Tlje above, taken from Grotelueschen (1980), is a gross 
compilation. However, a careful analysis shows that these 
descriptions^Sf-evaluation differ chiefly in the respect already 
mentioned— either -tHey relate to program delivery or to program 
operations (both institutional apd within the social context). 

The above definitions imply varying purposes and goals 
[or evaluation but also suggest distinct approaches. As noted, 
planning programs and measuring their impact and efficiency are 
significant operational concerns of administrators, whereas 
assessing participant performance, teaching methods, techniques, 
and materials are the specific concerns of those responsible for 
improving program delivery. These dual sets of concerns are of 
■import not just to administrators but to policy makers and 
educational planners. Indeed, evaluation exists largely because 
social policy makers and planners require the data (Williams, 
1972). * * 

A POLICY PROGRAM AMD EVALUATION DIAGRAM 

As an institutional process, program development is 
equivalent to program planning (Cook, 1971). In this sense, it 
included: (1) general policy considerations, (2) overall program 
operations, and (3) the realization ot teaching/learning, or 
education activities per sc (Rivera, Patino, and Brockett, 1982). 
Each of these functions— policy administration, institutional 
operations, and program realization — is performed usually by 
specific, differentiated personnel within an educational institu- 
tion,, although in small organizations with only one or two pro- 
grams these function^ sometimes overlap. 



^ The three tiers just described Comprise, as it were, the 
' major portion of a truncated pyramid, the top of which— 
whether conceived as attached or separate from the pyramid 
base— represents policy' making and planning. Thus' an image 
resembling that on the reverse of the U.S. dollar bill is concep- • 
Kiahzed, a nqvus ordo seclorum, with the policy /planning "eye 
of God" at the top. The three program tiers in the truncated 
portion of the pyramid include policy administration, overall 
program management, and delivery of specific programs (see 
Figure 1). 

With the main construct in place, it is a simple step to . 
recognize operational concerns, £s having to do with the two 
tiers comprising policy administration* and' program management. 
The bottom tier, which refers to delivery of programs, is con- 
cerned with classical evaluation related to educational materials 
content, instructional methodology,^ and learner performance 
(Popham, 1972; Scnven, 1967). 

Assuming then that the institutional process, as it refers to 
policy administration and Overall program management/is 
primarily directed toward improving operations, it becomes clear 
that administrators will be interested in 'evaluative processes 
relating in principle to planning, impact, and efficiency ortost- 
ettectiveness. Insofar as they are concerned with program cur- , 
nculum and delivery, their interests will be in participanf 
achievement, teacher methods and techniques, and the utility of 
materials. Thus, evaluations may aim at either improving pro- 
gram operations. flr program curriculum, while recognizing the 
relationship betweenftlte two. Nevertheless, Comprehensive 
evaluation is rare, and more often one or another type of. 
evaluation is undertaken at any one time. r - 

Once'an evaluation is completed, then the question arisefc 
as to its utilization— a question which, while crucial, is beyond 
the scope of this paper. For the present purpose, it is assumed 
the evaluation willy serve to improve operations and/or Cur- 
riculum, and jn certain cases it will be carried further in its flq>v 
(feedback) to influence policy and planning for the educational 
program. Thus, Figure 1 indicates evaluative feedback both to 
improve the program (including program delivery and opera- 
tions) and to aid in the formulation (including change) of policy. 

Figure 1 is an idealization. The actual policy/planning, 
program development, 'and evaluation flow is shaped by many 
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different contingencies; it cannot be limited to any' one set of 
events nor be represented by ajinear arrangement. Nonetheless, 
the diagram may serve to clarify certain generalities about the 
process and make the second p^rt of this- paper clearer. 

INTERNATIONAL STANCES TOWARD EVALUATION IN 
EDUCATION FOR ADULTS 

In 1968, a UNESCO literacy specialist, (Saksena) took a 
highly positive view toward evaluation.- Referring to the Ex- 
perimental World Literacy Programme (EWLP), he wrote: 

4 

, Evaluation as a supporting service is now generally well ac- 
cepted as, an essential component of scientific planning anci/ 
<. 'programming. w * 

Another UNESCO specialist, however, in remarking on the * 
use, or lack of. use, of folk media in mass media, took the.op- 
posite stance in 1974, emphasizing the. dearth of research and 
evaluative studies (Mathur). 
* ' In the same year, the 'World Bank recognized the need for 
evaluation and, indeed, its two-fold purpose of operational and 
curriculum improvement. The following statement appeared in 
its Education Sector Working Paper (December 1974): 

Jhe creation qf machinery for regular evaluation is essential 
for the effective management of education, as it is the main 
channel through which research and. development can be 
introduced into decision making. Evaluation also con- 
tributes'to the better design of educational schemes by re- 
quiring a de&r, .operational formulaticwi of their objectives. 

In 1975, John Lowe, a specialist for tne Organization for 
Economic Cooperation and Development (OECD) and former 
Professor and Chairman of Adult Education at the University of 
Edinburgh, wrote in his UNESCO-published The Education of 
^fiults: A World Perspective: 

Within recent years the literature of adult education has 
'been crammed with enjoinders about the cardinal necessity 
of evaluating programmes in a systematic fashion for the 
sake not only of improving* internal efficiency but also of 
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showing to sceptical officials in government departments 
that adult education institutions are competently managed 
and worth financing. / 

Lowe then added: 

Yet* there can'be little doubt that the overwhelming majori- 
ty of institutions make no convincing attempt to assess jhe 
effectiveness of their programmes, even in terms of their 
own objectives. Still less is there any attempt to calculate 
social or economic benefits. 

In 1976, UNESCO's supreme legislative body, the General 
Conference, at its 19th Session held in Nairobi, Kenya, adopted 
an international Recomtnendatidn on the Development of Adult 
Education. This document in section IV, point 36, states: 

Systematic evaluation of adult education activities is 
necessary to secure optimum results from the resources put 
, into them. For evaluation to be effective it should be built 
into the programmes of adult education at all levels and 

stajges. • 

The era of big evaluation*^ adult education had officially 
arrived, but in fact had really begun much earKer. The EWLP 
was conceived of in 1965 as a major evaluation research experi- 
ment as well as a new program concept— functional* literacy. 
So much for the steady development of evaluative research 

> in large-scale, programs. Still, there was another concern with 
respect to evaluation, and it was best expressed in another 
UNESCO document, Learning to Be, sometimes called the Faure 
Report after the former French Minister of Education, Edgar 
Faure, who chaired the committee responsible for the undertak- 
ing. That concern was with the original meaning and purpose of 
evaluation— recalling its evaluative nature rather than its 
significance for validation or decision making. In part 3, the core 
xff the volume, principle 11 speaks to the issue of access to dif- 
ferent types of education and professional employment. This 
principle argues that "access should depend only on each in- 
dividual's knowledge, capacities and aptitudes, and should not 
be a consequence of ranking knowledge acquirecf in school 

^ above or below experience gained during the practice of a pro- 
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tession or in pnvate studies" (UNESCO, 1972). The statement is 

clarified as follows: . * 

> ^ «. 

Examinations should serve essentially as a means of com- , 
paring skills acquired under varying conditions by in- 
dividuals of different origins, a mark not of a conclusion 
' but of a .starting-point, helping each individual to assess the 
effectiveness of his own study methods. Evaluation pro- , 
cedures should measure mi individual's progress % asjnuch as 
the extent to which he conforms to externally fixed stan- 
dards, (p.' 204; italics mine) 

Thus, the document speaks out for something other than t 
either norm-referenced or criterion-referenced evaluation. It calls 
tor an evaluation ot individual progress in the educational set- 
ting.' This sentiment is echoed in 1974 by the Government of In- 
dia in its Non Formal Education report. In reviewing a scheme 
to educate young adults in thje J5-25 age group in literacy, the 
report states: w - v 

Instruments for evaluation will have to be of two kinds, 
one for evaluating the progress of the adults, and^one for 
evaluating the success and impact of the programme, (p. 10) 

Thus, international stances toward evaluating education for 
adults appear to differ in several respects; there are: 

. a) positive claims that evaluation is alive, well, and doing 
its job in international arenas where education for adults is 
fostered; 

b) disclaimers that evaluation is doing what it ought to be 
doing and is as aliv£ and well as some would suggest, at least in 
certain areas of concern; 

c) enjoinders for developing evaluations at all levels and 
stages ot educational programs that serve adults; arfd 

d) caveats about the purpose of evaluation, reminding 

- educational policy makers and planners that learning is a matter 
of progress and not just of meeting some criterion; and further, 
that evaluation is of various kinds— operational and curricular. 

The last category of caveats is particularly- significant 
because it tinderscores the importance of stepping back to con- 
sider the final purposes of evaluation as they relate to the 
criteria underlying educational policy making and planning. 

EE£ • • / lb 
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CRITERIA QF EVALUATION IN 
, EDUCATION FOR ADULTS 

, THE PO.L1CY-EVALUATION RELATIONSHIP 

Program development and evaluation are outgrowths 
• usually of policy mandates. While a truism' this principle v 
deserves reiteration because it underscores th*it evaluation is not 
the practice of collecting information for its own sake. Professor 
Rodrtey Skager of UCLA (1978) claims that educational evalua- 
x tio 0\j^ always undertaken in order to facilitate decision-making 
or policy formulation" (p. 26). He distinguishes educational 
research from edu<!ational evaluation, remarking that research is 
often conducted to contribute to general .knowledge rather than 
\ to any particular decision needs. "In contrast/' Skager writes, 

"evaluation should always be guided by concern for how the in- 
formation is ultimately to be used and for what purpose it is to 
be used" (p. 26). Since evaluation involves deliberate expenditure 
of time and resources that might otherwise go directlyrinto the 
teaching and learning process, if must have a strongly utilitarian 
orientation. 

In The Evaluation of Literacy Programmes (Couvert, 
^ 1979), the evaluation of a project is seen, to have two fixed 
^points: 

a point of -departure determined by the initial -so.cial, 
economic and cultural situation of the prospective par- 
ticipants in their natural milieu; and a point of completion, 
i.e./ the ultimate situation as conceived and desired by the 
authorities responsible for deciding on literacy action. 
(p. 62; italics mine) 

Thus, evaluation is both the result of policy concerns and 
the object of its action, at least in part, for as Skager; states; it is 
always undertaken* in order to facilitate decision-making or 
policy formulation. But like education planning, evaluation is 
more than just a part of the policy-making process; it often 
, serves to change policy. And like planning, evaluation comprises 
an irritant function, aiming at changes in programs and systems 
(Eide, 1964). Nevertheless, changes in policy may be seen as part 
of the policy-forrfmlation process, and the intimate relationship, 
even circularity, of policy making and evaluation is apparent, s 
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THE PLAIHN1NG-EVALUATI0N SYMBIOSIS 

EducationaKplanningjs only one' of 'the possible influences 
on policy makers and policy making— often, purely political 
considerations determine .use of resources, but, planning .has ex- 
perienced an /astonishing rank£ development" since W.W. II, 
(Eide, 1904,/p. 70). The trerfd has beeri toward' accepting ,educa- 
' tional planning as a continuous government" responsibility aimed 
* at. a) the need for rapid adaption of government policies to 
changing conditions; b) the demand for* greater coherence in the 
, formulation of government policies; and c) increased use of 
research as an instrument towards mare rational government 

* policies (Eide, p. 70). - 

There is increasing use of evaluative and other applied 
research at governmentaf levels/ despite conceps with lack of" 
evaluation utilization (BOruch and Wqrtman, 1981; Ciarlo, 
,1981). As educational' planning has developed, the need for data 
^ *to confirm or change planning formulas becomes crucial; deci- * 
" sion> making is a main characteristic of -planning as it is for > 
evaluation (Bhola, 1979; -Spaulding,. 1974, 1982). Jn a paper 
presented at the Annual Meeting of the Comparative and Inter- 
national Education Society (1982), Spaulding makes note of this 
"close symbiosis between. evaluation $nd planning' (p. 1). 
The significance of # the symbiosis* between planning and 
/evaluation is rendered. more clearly when one considers the 
criteria used for ascertaining the educational needs of .society and 
how these criteria are translated interspecific recommendations 
for the level and structure of expenditure on the educational 
system. Professor of EcQnomicsfHerbert S. Parnes of Ohio State 
University, in examining approaches to the assessment of educa : 
(ion needs, summarized thereinto five catagories: (1) social de- 

* mand, (2) returns to educatidrv (3) econometric models, (4) 
irjtopewr requirements, and (5) cultural requirements (1964, p. 

53 ff.). 

While it is not the point pf this section to elaborate Qn 
these approaches, each' of them suggests differing criteria and-^ 
therefore differing methods of planning and ultimately of what 
will be evaluated, for evaluation ftV th£ main is the measurement 
of attainment of objectives— whether operational or curriCular. 

The two approaches to educational planning that have the 
most significance for evaluative, research are the manpower ap- 
proach and the cultural approach. While all approaches in a 
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thoroughgoing plan may be utilized to a greater or lesser extent, 
\during the past two decades the manpower approach (wit^ome 
Competition from the rates-of-retuni approach) has dominated. 
'In 1^60, the OECD, in its Mediterranean Regional Project, 
assisted in educational planning for the governments of Greece, 
Italy, Portugal, Spain, Turkey, and Yugoslavia, attempting a * 
brWl-based planning operation (Fames, 1964, p. 63). Attention 
wis given in varying degrees both to manpower .and .broader 
cultural objectives, and to quantitative and qualitative factors in 
formulating recommendations for expanding and improving the * 
educational system. Nevertheless, emphasis remained with man- 
power concerns! * t * » « 
^ In arguing for the'use of a manpower approach as a plan- 
ning device, Parnes procedes as follows; ' j 

In the manpower approach, one postulates a given rate and 
character of economic growth and asks what investment in 
education is necessary to achieve that growth objective. 
The cultural approach, on the other hand, stresse^educa- 
tion as a social "investment" ^to, which returns cannot be 
t calculated in money terms— an investment in values that - 
are either indispensable or highly desirable to the^society, 
e.g., ap informed citizenry, eqilality of opportunity -etc 

• It fallows, therefore, that short of educating everyone up to 
his capability . , there is no way of specifying educa- 
tional needslln ^ny absolute sensa Society needs as much 
education as t^$@ble and wjllingVo pay for. The decision 
isMnexorably a political one; 'and the best the planners,can 
' do is to indicate the cost implications of alternative -policy'; 
choices, (p. 60) 

Pames then adds a consideration that has ( certain implications 
for education specialists; he says: . 

,^It m ay be that these comments overstate the differences in 
^•kthe extent to which .manpower aijd cultural objectives, can 
"yield unique estimates of educational needs. It may be, for 
example, that the amount and typcof education nec^sary 
to produce a "qualified" citizen is just as ascertainable as 
- the amount and type pf education necessary to produce a ' 
qualified engineer. But if this is so, there is certainly not 
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rfte same consensus in the former case as in the jatfer. In. 
any event, 1 confess that 1 am unable to conceive a set of 
operations in the cultural .approach analogous to those that 
havs been set forth above for the manpower approach, 
(p. 61) 

It is no surprise then that educational programs have* come 
to emphasize qualified engineers over qualified citizens, since 
qualified citizens cannot be measured in economic, quantifiable 
* terms. And even at the basic levels, functional (work-oriented) 
literaty takes the place of traditional formats. The symbiosis of 
planning and evaluation is firmly established and can be further 
seen in the following brief review of the Experimental World * 
Literacy Programme (EWLP). 

THE EXPERIMENTAL WORLD LITERACY PROGRAMME 

Since. the 1960s, governmental and intergovernmental 
organizations have intensified their planning and expanded their 
policies for program activities in various fields. Intergovernmen- 
tal operations in the 1970s and early 1980s ^re especially' im- 
pressive in that large-scale operations have built-in, large-scale 
"Strategic evaluative rese^ch efforts. Currently Andei^vay is the 
' World Health Organization's "Health for All by) the Year 2000" 
program, f which incorporates global strategies for evaluation 
(WHO, 1982). But ceftainly in education the outstanding cross- 
cultural program incorporating large-scale evaluative research is 
the UNESCO/UNDP Experimental World Literacy Programme 
(EWLP). In 1979, the International Development Research Center 
(IDRC) in Ottawa, Canada, recognized it as "a first in research 
and evaluation" but added that "even the cumulative efforts of 
the EWLP have not created the base for rriajor decisions about 
literacy" (p. 6). 

Until fecently, literacy has been primarily a sphere of ac- 
tion rather than analysis, and it would appear that the EWLP 
has created the base for some major decisipns— not the least_ of 
which are 4 contained in the IDRC review of policy, research and 
action in favor of literacy— such as the powerful role of plan- 
ning and long-term goals, the significance of organization and 
administration, ,and th$ major factors in the achievement of 
-literacy, which are: (1) the principle of national commitment; 



(2), the principle of popular participation, and (3) the principle of 
coordination. 

It has been learned that for a country to silcceed in achiev- 
ing its economic social, and political goals, the achievement of 
literacy should be a stated and supported policy of a national * 
plan. The value and objectives of a national literacy program 
must be viewed by the target group as being relevant and useful 
to them and their community. And the administrative tasks in 
implementing a national policy must involve various ministries', 
in&rttitions, industrial enterprises, trade unions, government 
e organizations, and individuals (Couvert, 1979, pp. 64-65; IDRC, 
1979, pp. 12-13). 

In addition, the planning bases for the program clearly in- 
dicate the priority given to functionality as an aspect of man- 
power development and socio-economic a4vancement. In The 
Evaluation of Literacy Programmes. A Practical Guide (Couvert, 
1979, pp. 48-53), some 29 basic evaluation indicators are^ro- 
posed for use in literacy projects. These indicators are catego- 
rized under the following seven headings: 1 * 

1. turnover in programs 

2. acquired skills 

3. economic change 

4. attitudes toward education 

5. vocational/occupational skills ^ 

6. use of mass communication skills 
7* health, hygiene and safety 

The largest number of indicators fall under heading 3, in- 
dicators oPeconqmic change, which reflects the stress on produc- 
tivity and the planning goals of manpower development, both -of 
which are clearly stated in the central hypothesis of the pro- 
gram, to wit: 

In favourable and well-ordered socio-economic conditions, 
a training process focussed on development objectives and 
problems provide^ the individuals concerned with the in- 
tellectual and technical means for becoming' more effective 
agents in the process of socio-econorfiic development, 
(p. 25) • 
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The above central' hypothesis in turn reflects the functional, 
work-related nature of the EWLP and clarifies why one-third of 
its evaluation indicators focus on such components as: 
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—growth of output per inhabitant 
—product quality * 
sale prices % 

|*nit costs of production m % 

•changes in volume of durable goods 
—changes in net global monetary income 
—additions to equipment for production, maintenance, and 
transport 

—changes in socio-economic attitudes and in the concepf of 
the role of individuals in society (pp. 51-52) , 

Evaluation of a functional literacy project amounts to 
verification of the central (planning) hypothesis set out above. 
Ultimately, the purpose. is to measure the efficacy of program 
methods, monitor the program internationally, and justify the 
expenditures— within the framework of manpower development 
and socio-economic change. 

REVIEWING THE SCOPE AND GOALS OF 
EDUCATIONAL PLANNING 

International cross-cultural projects are indeed generally 
dominated by the application, to education (and especially to 
education for adults) of the assumptions used to evaluate pro- 
ductivity of economic enterprises. The reason for this lies in no- 
tions of rationality. Productivity is often equated wM,njtionality 
and indeed the educational systems and adult literacy^projects of 
participating countries are ranked by UNESCO in terms of theij* 
relative rationality— that is, in terms of productivity. 

* The economic, rational-man view of education is both ex- 
citing and depressing intellectually— exciting because it links 
adult literacy, for example, to functionality and in particular to 
work-oriented functionality. But it is depressing because it ig- 
nores other educational values and aims. If education is to be % 
considered as primarily a tool for investment felated to work 
productivity, then education becomes obviously inadequate in 
front of the inequalities that beset .societies. Education for adults, 
I would argue, suffers when programs concentrate only on their 
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learning >to be "more effective agents in the process of^socio- 
economic.development" (Anderson and Bowman, 1964, p. 9). 
Close analysis shows a people and their education to be m6re 
than merely part of the process of socio-economic development. 
IndeedNl would argue that soao-economics is, and should be, 
only part of the larger process'of human development despite 
the difficulties ol measuring cultural gains. . * • 

• C. Arnold Anderson, Professor of Education at\d * ' 
Sociology, and Mary Jean Bowman, Research AsSociaTe Pro- 
fessor of Economics, both of the University of Chicago, in ex- 
amining the scope and goals of educational planning, cfefine 
educational planning as "the process of preparing a set of deci- 
sions for future action pertaining to education.-" They add: 

But this is only an initial step toward delineating the 
theoretical foundations of education planning. What is 
regarded as Is^ucation ' varies, and those variations are 
central to any assessment of the bases or the implications* 
of educational planning. .9) * 

Having made that point, Anderson and Bowman m3fcc yet 
another even more important statement reflecting the value-laden 
quality of education and planning and, by extehsionfevaluation* 
They write: 

It is essential in the first instance to distinguish two very 
different situations. We can—and this is usual— treat 
educational planning as an adjunct or subhead of general 
economic.planning. Or we can deal with educational plan- 
ning - *^ its own right, with economic elements taken only 
as an aspect of it. In the first case educational planning 
derives from, or more correctly, constitutes merely an ex- 
tension of, manpower planning. This approach reflects an 
orientation to planning of production and employment, and 
the goal becomes manpower production, (p. 9) 

> 

This is the approach that has dominated educational plan- 
ning since the 1960s. Anderson and Bowman make a further 
comment, which is almost revolutionary in intimating that the 
power of thinking for themselves might be returned to 
.educators. Again, I quote: 
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When the aims and operations of education are considered 
in their own rigftt as a focus of planning, the aim can be 
as manifold and complex as the functions education is ex- 
pected to perform. Manpower^onsiderations become mere- 
ly one aspect of educational planning with no necessary 
priority over oth^r goals. The focus comes to be more or 
people, less on production of "human resources-." (p. 10; 
italics mine) 

UNESCO has made an enormous contribution to the world 
literacy effort, not so much in numbers of participants served 
but in terms of research. It has contributed to an understanding 
'what a truly cross-cultural analysis of educational programs 
can be, and especially with regard to evaluation. For this, if for 
nothing else, it deserves applause, appreciation, and support. At 
the same time it provokes consideration of other issues for it ex- 
emplifies par excellence what is happening in the world of 
education today: the turn toward productivity standards to 
define educational aims. 1 argue that this orientation needs to be 
brought more into the open for consideration by educators and 
society as a whole. 

REVIEWING THE SIGNIFICANCE Q^COST STATISTICS 

This paper suggests that educational planning for man- 
power development needs to be reviewed because it is limiting; 
hy contrast, an integrated educational plan balances functionality 
with cultural concerns. In this section a second claim is made, to 
the effect that cost statistics are not adequate for judging the 
worth of adult literacy or, far that matter, any other educational 
endeavor. The principal argument is drawn from the Cost- 
Effectiveness Report on the Work-Oriented Adult Literacy Pilot 
Project in Iran (Smyth, 1972). 

Smyth arrives at surprising conclusions regarding questions 
of cost-effectiveness, and in particular with respect to the long- 
term versus short-term significance and impact of adult litera 
He writesf • ' * 




Grounds for concluding that the (EWLP) project wa$ an 
economic failure (or success) simply do not exist, amd prob- 
ably cannot be established empirically anyway. The most 
lEtfat can be concluded is whether the project's authorities 
<^ followed correct economic principles, (p. 76) 



Cost statistics are not enough, as the International 
Development Research Center (IDRC) also notes in reviewing 
the above report (1979, p. 81*). Indeed, Smyth's argument gives 
pause tor thought about cost-effectiveness evaluation in educa- 
tion generally and renews the perennial consideration of equity 
and efficiency as conflicting goals with respect to education. 

In his report, Smyth also takes sides against the notion of 
integrating literacy with other development activities: 

[0]f course, integration can be attained by restricting pro- 
grammes to places of work, given on-the-job ^ with incen- 
tives and sanctions to encourage attendance, but that rather 
„ ignores the great mass of illiterate peasants and workers, 
m£n and women who do not earn, a living in factories or 
large workshops. And anyway, properly "integrating" a 
projecj^with other developmental activity is no assurance of 
greater economic benefit, (p. 70). 

Smyth goes on to make this point: 

From a purely economic standpoint, it is not a priori 
significant whether a programme is general or specific; 
what matters is that the .rate of return on investment in it 
should be high enough, Which may or may not depend on 
the degree of generality or^specificity. And simply because 
a program is given on-the-job . . . [does not mean] that it 
is more economically successful than a general programme 
given in the evening in a rural primary school to a mixed 
bag of peasants and children, (p. 170) 

Smyth's comments raise a number of conundrums. One is 
the question- just mentioned, that of equity versus efficiency. 
Whereas equity is a goal, efficiency is a rationality concept. As 
educational researchers Anderson and Bowman (1964) point put, 
"efficiency is a rationality concept that implied getting the most 
out of the least, whatever the nature of the rewards or ends 
may be." But what criteria will be used? Will individuals be 
selected for further schooling based on how much additional 
productivity can be predicted for one versus another person? 
Will priority be given to groups or localities where proposed 
educational efforts will evoke the largest response in attendance 
and where demand for further schooling is greatest? Or, should 



19 - 



. investment in education tor adults be. made where the expected 
ratio ot gains in economic output to costs is highest? Tftese dif- ' ^ 
tenng criteria posed at length by Anclerson and Bowman in 1964 
are still pertinent, and Smyth's conclusions suggest that efficiency 
arguments should not be considered at all* except in measuring 
the efficiency of administration and management. 

The second major issue raised by Smyth s comments relates 
to educational planning and the objectives on which evaluations 
will be based. Essentially the two major approaches to interna- 
tional planning for educational development— manpower plan- 
ning and rate-of-retlirn— though different in various respects, 
view education as investment in human-resource development. 
In essence they both treat educational planning as an adjunct or 
subhead of general economic planning. Neither, in other words, # 
considers educational planning in its own right, with economic ✓ 
elements taken as only one aspect. However, in Smyth's case his 
rate-ot-return approach <joesn't keep him from recognizing the 
general good, or value, of education in and of itself, and he 
conceives ot the larger investment in human beings as more than 
an immediate or short-term 'issue of cost-effectiveness. 

The international concern with functionality, productivity, 
I and socio-economic indicators is understandable but it never- 

theless reflects a narrow approach to education. It skews the full 
meaning and significance of education, including education* for 
adults. Nevertheless, there is today a strong focus on cost- 
ettectiveness and efficiency and a trend in this direction appears 
likely tor the coming decade. Thus, the predominance of 
economic over other concerns in education— such as 'equity, par- 
ticipant progress, or even achievement—will probably continue 
unless this trend is v altered. * . 
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IMPLICATIONS FOR UNIVERSITIES, 
, PROFESSIONAL ASSOCIATIONS, ' 
AND ACADEMICS 

- • TRENDS IN TH| ERA OF BIG v SOCIAL SCfEfcCE 



In reviewing the emerging '%ra of big social science," Pro- 
fessor Walter Williams of the University.^ Washington noted 
that a significant increase in soundly conceived and executed 
evaluative studies requires that social agencies, as the 
primary developers and users of social prograih evaluative 
results, establish large, well-trained staffs with sufficient technical 

'•and administrative skills (1972). 

In reviewing the capacity of social science organizations to 
perform large-stale evaluative research, Williams further noted 
that, "sfs compared- with universities, nonuniversity research 

, organizations such as the Rand Corporation generally seem bet- 
teNable'ih <ln institutional sense to perform large-scale research, 
Jthe results of which are expected to have a direct effect on social 
agency decisions" (p. 306). It would seem that academic social 
science is not yet oriented toward serving as an "instrument" of 
state administration. 

Howard E. Freeman of UCLA and Mariam A. Solomon of 
the System Development Corporation (1981) observed that there 
is a relative decline in the number of dollar expenditures for 
evaluations undertaken by researchers- in universities. They 
maintain that the profit-making firms and a few aggressive non- 
profit groups more and more dominate the field from the stand- 
point of the actual conduct of studies. Competition for govern- 
ment contracts is fierce. Freeman and Solomon point out that 
the decline in university evaluative research is due in part to 
government and foundation contract procedures and the short 
turn-around times required from when requests for proposals are 
advertised and when bids from prospective contractors must be. 
submitted/ Also, the time allotted for contract performance 
oftentimes makes it impossible for academic groups to compete. 
Finally, tftey add, the commercial sector organizations and the 
"nonprofits" may do a job at less cost, sometimes because they 
cut corners, but more probably because they have smaller 
bureaucracies than universities and more incentives to be effi- 
cient. As to the quality of the .work carried on by profit-making 
firms as compared with universities, the authors suggest? that 
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while quality in the past was superior in the universities, today 
"the case is an open question" (p. 16). 

Freeman and Solomon suggest that the shift in the conduct 
of evaluations to the firm will influence the way the field 
develops methodologically. Methods are likely to be more prac- N 
tical than classical research procedures (p. 17). It appears that i 
leadership in evaluative research is moving to government agen- 
cies and nonuniversity policy research organizations and away 
from universities and their research centers. 

A major concern mentioned by Freeman and Solomon 
(p. 18) is the apparent lack of knowledge within universities 
about how to undertake and carry out successful evaluations. It 
would appear that the first task of the universities is to master 
the technical procedures before seeking to obtain or regain a 
leadership Tole in this area. 

What specifically are the purpose and role of universities, 
professtefial associations, and academics concerned with com- 
parative and international practices? Is the purpose to be an in- 
strument of the state? Is the role to be that of technical advisor, , 
consultant, arid occasional grantee? Is it to be critically analytic? 
Is there a leadership roje, which includes a political role, that the 
universities and individual faculty are willing and ready to 
assume? Is the answer: aljof the above? 

Of several major approaches to evaluation, three appear to 
dominate at the present time, systems analysis, behavioral objec- 
tives, and decision-making approaches (House, 1980). Of these, 
systems, analysis is most utilized within the framework of large- 
scale evaluations. As noted earlier, the emphasis of educational 
evaluation often enough is dn efficiency. The trend in the 1980s 
appears to be toward greater emphasis on efficiency and costs 
"flten on benefits and effectiveness. 

Cost-benefit and cost-effectiveness analyses have their roots 
in efforts of economists to explicate the national and inter- 
national consequences of major inputs; conceptual and 
technical developments are necessary in order to mak^the 
approach and methodology amenable to the services area. 
(Freeman and Solomon, 1981, p. 19) 

The trend toward efficiency evaluation may be inevitable^ 
even necessary, and educators may need to learn "to calculate 
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social and economic benefits/' as Lowe counseled in 1975 (p. 
104), But there is, nevertheless, cause for concern that academics 
will neglect their purpose and role to counter the prevailing cost- 
efficiency emphasis in large-scale evaluations. What should be 
their next steps? 

NEXT STEPS 

* First, cooperation and coordination among universities, 
perhaps in the form of consortia, would be a logical beginning 
step. Certain consortia already exist, within which strategies 
could be, developed for ways in which to cooperate and coor- 
dinate interests in comparative and adult education with 
evaluative research. Some consortia are government sponsored, 
such as the US/AID Regional Consortia for International 
Development. Others, such as the Southern Regional Education 
Board, are the outgrowth of mutual collaborations in other areas 
of concern. These existing consortia might serve to support new 
cooperative, efforts among universities with concerns for large- 
scale national and international program analysis. f * 
Secondly, cooperation and coordination among professional- 
associations is long overdue. The tendency at present appears to' 
be for each association to create separate units or committees in- 
ternally to deal with specific issues such as evaluation. At the 
same time, independent associations of specialists, such as the 
Evaluation Research Society and the Evaluation Network, are 
cropping up. It would seem appropriate for professional associa- 
tions with evaluation committees or coqcehis to seek to 
cooperate and even coordinate., activities with those associations 
specifically dedicated to one or another of these activities: adult 
education, comparative education, and evaluation. Networks 
among" associations would serve to encourage cofresiveness . 
among professionals with overlapping interests, and perhaps 
&rve as a means for developing specific projects that might nor * 
be feasible by one association alone. 

Thirdly, academics themselves need to seek out means of 
collaboration within the framework" of university consortia and 
professional association networks, and among colleagues under- 
taking similar efforts in comparative, adult-education evaluative 
* research. 

Fourthly, clearinghouses of information are needed to col- 
lect, analyze, and disseminate information among colleagues 
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about works in progress and intended evaluative research. To 
date, clearinghouses have tended to be insular in their concerns, 
serving only faculty within one university or-at best statewider 
A broader_perspective is needed; perhaps within the Educational 
Resources Information Center (ERIC) or through a -government- 
sponsored national center for evaluative research at the national 
and international levels. In this regard, it is perhaps worth 
noting that evaluation at different levels is distinct in concept, 
method, and consequences, and that a clearinghouse would want 
to limit its efforts to comparable research efforts. Furthermore, 
documentation centers that already exist need to make strenuous 
efforts to acquire the basic literature and especially program 
reports of evaluations at the international level. Documents on 
the EWIP, for example, are almost impossible to obtain in the 
United States; they are not even available from the U.Si/Com- 
mission for UNESCO in Washington, D.C. Even when the com- 
missions arid branches of international organizations do have 
some of the relevant documentation, access is difficult as they 
are hot functioning as lending libraries. 

Fifthly, interdisciplinary efforts to cooperate and coordinate 
thinking about evaluation are needed. Economists, sociologists,/ 
and educators need to discuss their differing criteria and / 
methods, with a view to complementing each other's researpn 
and contributing to new ways of operating, if that were to 
prove feasible. Furthermore, academics in these areas need fur- 
ther contact with policy, planning, and program development 
staff as well as with program evaluation researchers and interna- 
tional development personnel in the field. In the age of com- 
puterization, such communication would not seem to be out of 
the question. 

There are surely other ways for universities, professional 
associations, and academics to begin to seek renewed purpose 
and roles in the area of large-scale evaluative research, especially 
with regard to education for adults at the international level. 
The preceding suggestions are meant to stimulate tho_ught^and 
preliminary action. 




CONCLUSIONS 

>*' PERSPECTIVES 

Evaluation harbors many definitions, purposes, ^nd-criteria. 
Different evaluative approaches exist as a consequence of differ- 
ing perspectives of education— of what should be analyzed 
within and as a result of an educational program. This 
multidimensional nature of evaluation renders the subject 
complex. 

What seems to be agreed upon is the need for evaluation. 
Until recently, for example, literacy was primarily a sphere of 
action rather than analysis. While evaluative^ efforts may not yet 
provide the basis for major decisions in the field of literacy, a 
' * beginning has been made. Nevertheless, certain caveats exist, 
and they include the type of information to be gathered. 
Presumably, dail collection will parallel the objectives laid down 
by policy and planning, but then the premises of policy and 
planning also require consideration— especially when literacy is 
' t linked almost entirely with manpower-development needs and 

literacy programs are judged primarily on the basis of cost 
statistics. 

As educational evaluation has moyefi frorji considerations 
of participant achievement into the realm of Cost analysis, and 
there is some logic ift this move, it'appears that short-term 
economic considerations are outweighing long-term educational 
payoffs. While contemporary wisdom suggests that there is little 
likelihood of reversing' this trend, the present paper >argues for 
educational outcomes to be put on a par with or abjove concerns 
with expenditures. v 

At the international level, certain commentators maintain 
that in judging educational, especially literacy, programs it is the 
progress of the participant which is crucial and not merely 
achievement according to some norm or established criterion. 
This would seem especially true fot developing countries or 
poverty areas where people have had limited exposure to>nd 
concern for education, whether formal or nonformaL 

CRITERIA 

Governmental education policies, especially when based on 
planning approaches, tend to emphasize manpower development, 
i,e., the human being as an economic resource. Important 
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^though the $ole of economics may be for the individual and the 
society, there are cultural considerations— equality, access to in- 
* formation, historical awareness, etc.— that form part of any in- 
. tegrated educational effort. To subsume these QonCems or 
relegate them to low priority is to court disaster in the long run. 
Jr Providing people with the knowledge and skills necessary to ad- 
vance- the productive processes of the economy represents only * 
, 'one dimensipn of a society's need for education. Another need is 
political, to assure the level of enlightenment required for effec- 
tive and responsible citizenship. Still another is the historical 
. dimension: to promote an understanding in the populace of their 
historical roots- and where they sit on the branch of contem- t 
porary development. International understanding is yet another 
* consideration, as is the sociological dimension, as is the need for 
'self-development within modem society. ^ 

The question arises as to how to conceive of a set of 
operations'in these various domains to compare with those set 
forth for manpower development. /\s the behaviorists have • 
taught (Mager,^974; Popham, 1972), anything can be 

transformed iri*£ behavioral objectives. The real question is, 

* ,When will academics concerned with an integrated cultural ap- 
proach to education get around to countering the purely man- 
. power approach? 0 ♦ 

' In addition, the question of evaluation of program/expen- 
ditures, while necessary, needs to be considered within the larger 
framework of long-term educational goals and their payoffs. 
Adult basic education rates-of-retum studies (Arkahsas, 1981) in- 
dicate significant economic gains for participants over their 
lifetimes and, therefore, for society, despite what appear to be 
majoF initial economic expenditures. Evaluation based on educa- 
tional olitcomes can be seen as a long-term marriage, so to 
speak, whereas that based solely on economics is a short-term 
affair with limited perspectives of the future. 

IMPLICATIONS 

The indications of the. present discussion are several. A 
review of ^he^values underlying educational policy and educa- 1 
tional planning is needed. A challenge to the en£consed 
manpower-development model is also needed, if only to 
underscore the multidimensional nature of education. 
^ % Evaluation is inevitably value laden, and approaches to it 
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define its final meaning. While both operational and curricular 
concerns confront the policy maker, planner, and program 
developer, priorities must be determined that underline broad- 
based educational concerns with individual progress and achieve- 
ment in various arenas oHcnowledge, skillsf attitudes, and 
aspirations. - ^ • 

1 Also, the universities, as well as professional associations 
and the academics who belong to both, are being squeezed out 
of the large-scale evaluation game, giving the lead to govern- 
ment agencies and private firms. The time is ripe for academics 
to clarify their purpose and role in large-scale evaluative efforts 
and to consider how their institutions and associations might en- 
sure the kind of cooperation and coordination needed for them 
to be able to assume ah active part. In this, the era of big social - 
science, such' is the task at hand if the current trend in 
efficiency-based evaluation is to be reversed. 

\ ■ ■ ■ 
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